What we do (not) know about the mechanisms underlying adaptive speech perception: A computational framework and review.

Speech from unfamiliar talkers can be difficult to comprehend initially. These difficulties tend to dissipate with exposure, sometimes within minutes or less. Adaptivity in response to unfamiliar input is now considered a fundamental property of speech perception, and research over the past two decades has made substantial progress in identifying its characteristics. The mechanisms underlying adaptive speech perception, however, remain unknown. Past work has attributed facilitatory effects of exposure to any one of three qualitatively different hypothesized mechanisms: (1) low-level, pre-linguistic, signal normalization, (2) changes in/selection of linguistic representations, or (3) changes in post-perceptual decision-making. Direct comparisons of these hypotheses, or combinations thereof, have been lacking. We describe a general computational framework for adaptive speech perception (ASP) that-for the first time-implements all three mechanisms. We demonstrate how the framework can be used to derive predictions for experiments on perception from the acoustic properties of the stimuli. Using this approach, we find that-at the level of data analysis presently employed by most studies in the field-the signature results of influential experimental paradigms do not distinguish between the three mechanisms. This highlights the need for a change in research practices, so that future experiments provide more informative results. We recommend specific changes to experimental paradigms and data analysis. All data and code for this study are shared via OSF, including the R markdown document that this article is generated from, and an R library that implements the models we present.

Thinking probabilistically in the study of intonational speech prosody.

Speech prosody, the melodic and rhythmic properties of a language, plays a critical role in our everyday communication. Researchers have identified unique patterns of prosody that segment words and phrases, highlight focal elements in a sentence, and convey holistic meanings and speech acts that interact with the information shared in context. The mapping between the sound and meaning represented in prosody is suggested to be probabilistic-the same physical instance of sounds can support multiple meanings across talkers and contexts while the same meaning can be encoded in physically distinct sound patterns (e.g., pitch movements). The current overview presents an analysis framework for probing the nature of this probabilistic relationship. Illustrated by examples from the literature and a dataset of German focus marking, we discuss the production variability within and across talkers and consider challenges that this variability imposes on the comprehension system. A better understanding of these challenges, we argue, will illuminate how the human perceptual, cognitive, and computational mechanisms may navigate the variability to arrive at a coherent understanding of speech prosody. The current paper is intended to be an introduction for those who are interested in thinking probabilistically about the sound-meaning mapping in prosody. Open questions for future research are discussed with proposals for examining prosodic production and comprehension within a comprehensive, mathematically-motivated framework of probabilistic inference under uncertainty. This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Language.

Encoding and decoding of meaning through structured variability in intonational speech prosody.

Speech prosody plays an important role in communication of meaning. The cognitive and computational mechanisms supporting this communication remain to be understood, however. Prosodic cues vary across talkers and speaking conditions, creating ambiguity in the sound-to-meaning mapping. We hypothesize that listeners ameliorate this ambiguity in part by learning talker-specific statistics of prosodic cues. To test this hypothesis, we investigate the production and recognition of question vs. statement prosody in American English. Experiment 1 elicits productions of questions and statements from 65 talkers to examine the distributional statistics characterizing within- and cross-talker variability in these productions. We use Bayesian ideal observer models to assess the predicted consequences of cross-talker variability on listeners' recognition of prosody. We find that learning of talker-specific distributional statistics is predicted to facilitate recognition, above and beyond what can be achieved via commonly assumed normalizations of prosodic cues. Experiment 2 tests this prediction in a comprehension experiment. We expose different groups of listeners to different prosodic input statistics and assess listeners' recognition of questions and statements both prior to, and following, exposure. Prior to exposure, ideal observer-derived predictions based on Experiment 1 provide a good qualitative fit against listeners' recognition of prosodic contours in Experiment 2. Following exposure, listeners shift the categorization boundary between questions and statements in ways consistent with learning of talker-specific statistics.

Online pragmatic interpretations of scalar adjectives are affected by perceived speaker reliability.

Linguistic communication requires understanding of words in relation to their context. Among various aspects of context, one that has received relatively little attention until recently is the speakers themselves. We asked whether comprehenders' online language comprehension is affected by the perceived reliability with which a speaker formulates pragmatically well-formed utterances. In two eye-tracking experiments, we conceptually replicated and extended a seminal work by Grodner and Sedivy (2011). A between-participant manipulation was used to control reliability with which a speaker follows implicit pragmatic conventions (e.g., using a scalar adjective in accordance with contextual contrast). Experiment 1 replicated Grodner and Sedivy's finding that contrastive inference in response to scalar adjectives was suspended when both the spoken input and the instructions provided evidence of the speaker's (un)reliability: For speech from the reliable speaker, comprehenders exhibited the early fixations attributable to a contextually-situated, contrastive interpretation of a scalar adjective. In contrast, for speech from the unreliable speaker, comprehenders did not exhibit such early fixations. Experiment 2 provided novel evidence of the reliability effect in the absence of explicit instructions. In both experiments, the effects emerged in the earliest expected time window given the stimuli sentence structure. The results suggest that real-time interpretations of spoken language are optimized in the context of a speaker identity, characteristics of which are extrapolated across utterances.

Information Integration in Modulation of Pragmatic Inferences During Online Language Comprehension.

Upon hearing a scalar adjective in a definite referring expression such as "the big…," listeners typically make anticipatory eye movements to an item in a contrast set, such as a big glass in the context of a smaller glass. Recent studies have suggested that this rapid, contrastive interpretation of scalar adjectives is malleable and calibrated to the speaker's pragmatic competence. In a series of eye-tracking experiments, we explore the nature of the evidence necessary for the modulation of pragmatic inferences in language comprehension, focusing on the complementary roles of top-down information - (knowledge about the particular speaker's pragmatic competence)  and bottom-up cues  (distributional information about the use of scalar adjectives in the environment). We find that bottom-up evidence alone (e.g., the speaker says "the big dog" in a context with one dog), in large quantities, can be sufficient to trigger modulation of the listener's contrastive inferences, with or without top-down cues to support this adaptation. Further, these findings suggest that listeners track and flexibly combine multiple sources of information in service of efficient pragmatic communication.

Predictability of meaning in grammatical encoding: Optional plural marking.

The markedness principle plays a central role in linguistic theory: marked grammatical categories (like plural) tend to receive more linguistic encoding (e.g., morphological marking), while unmarked categories (like singular) tend to receive less linguistic encoding. What precisely makes a grammatical category or meaning marked, however, remains unclear. One prominent proposal attributes markedness to the frequency or predictability of meanings: infrequent or less predictable meanings are more likely to receive extra linguistic encoding than frequent or more predictable meanings. Existing support for the predictability account is limited to correlational evidence, leaving open whether meaning predictability can cause markedness patterns. We present two miniature language learning experiments that directly assess effects of predictability on morphological plural marking. We find that learners preferentially produce plural marking on nouns that are less probable to occur with plural meaning-despite the fact that no such pattern was present in learners' input. This suggests that meaning predictability can cause the markedness patterns like those that are cross-linguistically observed.

Effects of distributional information on categorization of prosodic contours.

Although prosody clearly affects the interpretation of utterances, the mapping between prosodic representations and acoustic features is highly variable. Listeners may in part cope with this variability by adapting to distributions of acoustic features in the input. We examined whether listeners adapt to distributional changes using the construction It looks like an X. When pronounced with an H* pitch accent on the final noun and a low boundary tone, the construction supports an affirmative interpretation (e.g., It looks like a ZEBRA [and I think it is one]). Conversely, when pronounced with a L+H* pitch accent and a rising boundary tone, it suggests a negative interpretation (e.g., It LOOKS like a zebra.... [but it is not]). Experiment 1 elicited pragmatic interpretations of resynthesized 12-step continua with these two contours as the end points. In Experiment 2, one group of listeners heard items sampled from the most ambiguous region along the continua followed by affirmative continuations (e.g., It looks like a zebra because it has stripes all over its body) and items near the contrastive endpoint followed by negative continuations (e.g., It looks like a zebra but it is actually something else). Another group heard the reverse (i.e., ambiguous items with negative continuations and non-contrastive items with affirmative continuations). The two groups of participants subsequently derived diverging interpretations for novel ambiguous items, suggesting that prosodic processing involves flexible mappings between acoustic features and prosodic representations that are meaningful in interpretation of speech.

Pragmatic inferences in context: learning to interpret contrastive prosody.

Can preschoolers make pragmatic inferences based on the intonation of an utterance? Previous work has found that young children appear to ignore intonational meanings and come to understand contrastive intonation contours only after age six. We show that four-year-olds succeed in interpreting an English utterance, such as "It LOOKS like a zebra", to derive a conversational implicature, namely [but it isn't one], as long as they can access a semantically stronger alternative, in this case "It's a zebra". We propose that children arrive at the implicature by comparing such contextually provided alternatives. Contextually leveraged inferences generalize across speakers and contexts, and thus drive the acquisition of intonational meanings. Our findings show that four-year-olds and adults are able to bootstrap their interpretation of the contrast-marking intonation by taking into account alternative utterances produced in the same context.

Talker-Specific Generalization of Pragmatic Inferences based on Under- and Over-Informative Prenominal Adjective Use.

According to Grice's (1975) Maxim of Quantity, rational talkers formulate their utterances to be as economical as possible while conveying all necessary information. Naturally produced referential expressions, however, often contain more or less information than what is predicted to be optimal given a rational speaker model. How do listeners cope with these variations in the linguistic input? We argue that listeners navigate the variability in referential resolution by calibrating their expectations for the amount of linguistic signal to be expended for a certain meaning and by doing so in a context- or a talker-specific manner. Focusing on talker-specificity, we present four experiments. We first establish that speakers will generalize information from a single pair of adjectives to unseen adjectives in a speaker-specific manner (Experiment 1). Initially focusing on exposure to underspecified utterances, Experiment 2 examines: (a) the dimension of generalization; (b) effects of the strength of the evidence (implicit or explicit); and (c) individual differences in dimensions of generalization. Experiments 3 and 4 ask parallel questions for exposure to over-specified utterances, where we predict more conservative generalization because, in spontaneous utterances, talkers are more likely to over-modify than under-modify.

Is it or isn't it: listeners make rapid use of prosody to infer speaker meanings.

A visual world experiment examined the time course for pragmatic inferences derived from visual context and contrastive intonation contours. We used the construction It looks like an X pronounced with either (a) a H(*) pitch accent on the final noun and a low boundary tone, or (b) a contrastive L+H(*) pitch accent and a rising boundary tone, a contour that can support contrastive inference (e.g., It LOOKSL+H*like a zebraL-H%… (but it is not)). When the visual display contained a single related set of contrasting pictures (e.g. a zebra vs. a zebra-like animal), effects of LOOKSL+H* emerged prior to the processing of phonemic information from the target noun. The results indicate that the prosodic processing is incremental and guided by contextually-supported expectations. Additional analyses ruled out explanations based on context-independent heuristics that might substitute for online computation of contrast.

Zipfian frequency distributions facilitate word segmentation in context.

Word frequencies in natural language follow a highly skewed Zipfian distribution, but the consequences of this distribution for language acquisition are only beginning to be understood. Typically, learning experiments that are meant to simulate language acquisition use uniform word frequency distributions. We examine the effects of Zipfian distributions using two artificial language paradigms-a standard forced-choice task and a new orthographic segmentation task in which participants click on the boundaries between words in contexts. Our data show that learners can identify word forms robustly across widely varying frequency distributions. In addition, although performance in recognizing individual words is predicted best by their frequency, a Zipfian distribution facilitates word segmentation in context: the presence of high-frequency words creates more chances for learners to apply their knowledge in processing new sentences. We find that computational models that implement "chunking" are more effective than "transition finding" models at reproducing this pattern of performance.